Update: 7/26/12 Hootsuite has changed ow.li from a 302 redirect to a 301 redirect. Thanks, Hootsuite, for reading! (Link to comment from Matt)
Hang tight, folks! Before we get into the whooo-done-it part of this post, here’s a basic illustration of how URL shorteners work.
It’s pretty simple actually, on the surface.
- You need a short domain name (like bit.ly – which is possible because of the .ly extension)
- A random URL (or custom “vanity” if you choose) is created on the short domain like bit.ly/h1oiSn/ or bity.ly/example-url/ (both fake)
- A visit to that URL sends the user via 301 redirect or 302 redirect to the page it’s linked to.
Here’s a real example of a URL I recently shortened with Bitly:
.png)
See? NORMAL. And Bit.ly uses a 301 redirect which is also most liked, since 301’s pass link equity.
And the Bitly URL should (ideally) never rank above the final “real” URL.
Danny Sullivan’s post, although a bit outdated now, shows (of the shorteners from that time) which services use 301s and which use 302s. Note that he highly advises against the use of URLs with 302s. Remember that for later 🙂
If you want to examine more details before we begin, you can check out the wikipedia page on URL shortening. However, the point of this post is to analyze a particular SERP which returns a shortened URL instead of the normal URL.
For one, startupcity.org should probably rank #1 – implied brand search. But… that’s not quite it. See it?
Perhaps if I restate it.
No prize though I’m afraid.
Strange result credit: tweet by Rand.
There are two things wrong with this screenshot.
- An owl.li URL is ranking in the first page on the SERPs
- Startupcity.org is not ranking as well as you’d think, as this is definitely an implied brand search.
Needless to say, Rand’s tweet highlighting this strange behavior certainly led to the idea for this post.
Initial Hypothesis: Owl.li URLs are NOT normal
But what’s different about them? What’s causing one to rank on the first page?
Who’s at fault – Google, owl.li, or the site owner?
Let’s see if we can find out!! In the process, we’ll examine the implications of using owl.li and come up with some best practices that you can follow to avoid this happening on your site.
The first thing I’d like to invite you to do is to join me on a guided video tour of a bunch of screenshots I took while investigating this.
Oh boy, this is a multimedia extravaganza on the Moz blog! I decided to walk through all my bits of evidence in a screencast – to give you a quick overview of the investigative process behind this.
Video 1/2 – you’ll want to expand to full screen
Video 2/2 – again, full screen will look best!
Well now… those videos were fun, yes? Alright, let’s examine the biggest takeaways from this first look.
You didn’t want to watch the videos? (Sadface) OK. Here are the 5 most significant screencaps:
1. What’s the Redirect Path To the Final URL?
This naturally was my first question. How is owl.li executing the redirects? And what a fabulous opportunity to use this new redirect checker chrome plugin by Ayima. The redirect path of this owl.li URL is as follows:
It’s a 302 redirect followed by three 301 redirects. I know redirect chains are not the best thing in the world, so could this be causing the strange indexation? I file that question in my brain for later.
2. Does StartupCity Have Internal Links Sending Mixed Signals?
You should ALWAYS internally link to the current version of your URLs. This makes your site faster, it prevents things from breaking in the future, and allows only external links/bookmarks to be passed through redirects.
Sometimes internal links pointing to pages which then 301 can send mixed signals. Sure enough, there’s some of that happening on startupcity.org
However, I’m not entirely convinced this is causing Google to actually rank the owl.li page.
3. What Does Google’s Documentation Say About Redirects?
Google states pretty clearly that they can and do index 302’s.
I do find it interesting that Google doesn’t say either way if a 301 can be indexed.
4. Can I Get BOTH URLs To Return in a Search?
I switched around the keywords, and now we have both URLs ranking on the first page. Yes, the owl.li URL AND the startupcity.org URL are both on the first page.
5. How Do You Make an Owl.li URL?
You’ll see in the video that I had a few unsuccessful attempts at making an owl.li URL (had no idea, never done it before!). I finally figured out where owl.li links come from. I’m sure some of you already know, but as I am not a heavy HootSuite user, I had to look. (Note: this image is not in the video. So consider this a bonus? Unless you’re watching the video – then you’re missing out.)
- First put the link in the link field
- Before shortening click the gear (advanced)
- Select owl.li from the drop down
The following conclusions and recommendations are my personal thoughts – I would follow them myself and refer them to clients. However, I have to be completely honest. I’m not 100% sure why Google is ranking that owl.li result for [startupcity 100].
I do have some strong hypotheses and tips for site owners.
Owl.li Was Created as an Alternative to the .ly Domain
I finally found some record of when and why the owl.li shortener was put into place.
- Here’s the post directly from HootSuite – to sum up, they state to have initiated it because of concerns over the .ly top level domain and civil unrest in Libya (the country from which the TLD belongs to). Users of HootSuite were given the option of choosing their shortener in April of 2011
- Here’s an article from the Social Times – the article more or less sums up the move to owl.li for safety and all that.
Now I know WHY this was put into place, but still don’t know why they chose to use 302’s.
Also, they claim users can avoid the .ly TLD – this isn’t entirely true though, because everything still passes through ow.ly as a redirect.
Owl.li Sends You Through a 302 and Then 301 Redirect
Owl.li is not like other short URL services. It sends you through a 302 and then a 301.
This is perhaps the biggest takeaway of all, which is worth restating. Owl.li sends their links via 302, through ow.ly which then 301 to the final page. This, again, seems like a double whammy. Not only is the “double redirect” not necessary, the 302 means none of the link equity is being passed on to your page.
Again, I would like to know – why does HootSuite tell us owl.li is an alternative to ow.ly (to avoid the .ly) but then direct all URLs through ow.ly anyway?
Google’s Side – Possible Reasons/Implications of Shortened URLs In the Index
Redirects alone don’t prevent indexation – Google does not say they don’t index 301’s or 302’s. They state in the affirmative that they may index 302’s – and they don’t state anything either way about 301’s. So it does make sense as to why the shortened URLs are in the index. (Please note that Bing does not index 302 redirects, according to this article from 2010 – which is likely why there are almost no owl.li URLs indexed).
Ranking signal – They were likely indexed because they were tweeted. Perhaps further indication of Google using Twitter as a ranking signal? Index the short URLs and assign a page value to them (by number of tweets, inbound links, etc.)? There’s really no value for the user to have them in the index, right?
Destination pages had issues – The pages they pointed to had some internal issues – like the IP addresses not resolving, or the internal linking, or large chains of redirects.
A short URL ranking well Is an anomaly – However, the startupcity.org result was the ONLY one that actually ranked above its destination page. All the other ones may be indexed, but I have never seen a short URL ranking on page one before. (If you have I’d be interested – tweet me @dan_shure with a screencap).
Site Owners – Follow These Best Practices
Ultimately, I don’t blame the site owner in this case (or in many others). But there are some best practices to follow that can reduce the risk of these issues (or others) appearing in the SERPs.
1. Reduce and ideally eliminate chains of redirects
- Use the new tool by Ayima to check for redirect chains on a page by page basis.
- To check redirect headers a bit more intensely, use URI Valet (set to Googlebot for indexation questions).
- You can even use Screaming Frog SEO Spider to check for redirects.
To eliminate redirect chains; instead of Page 1->Page 2->Page 3
Use
Page 1->Page 3
Page 2->Page 3
2. Use noindex, the URL removal request, or robots.txt to permanently get a page deindexed
Google will index 301 and 302 redirected URLs. If the page still exists, use a meta noindex tag on the page and request removal with webmaster tools. If the page no longer exists, do the URL removal request and use robots.txt to keep it from getting back in the index.
3. Keep your site’s internal links updated.
As mentioned, this helps to;
- Make your site faster
- Eliminate mixed signals
- Prevent things from getting broken in the future
- Only external links/bookmarks have to get passed through redirects
Point your users at the current version of your URLs whenever possible. Anything from pointing to the homepage as /index.php when its just / – always keep your internal links updated.
Use Screaming Frog to crawl the site and look for any that need fixin’
4. Use Ow.ly in preference to Owl.li
We have seen that by using owl.li you’re STILL using Ow.ly – because the ow.li 302 redirects through ow.ly. Who knows if this will always be the case, but why send your links through an extra redirect and also risk the owl.li URL showing in the SERP of the destination page?
Right – I never did say 🙂
It sure would be most fun to point blame at one particular party – but in this case it’s completely a team effort.
- Google – is indexing thousands of owl.li URLs
- Owl.li – is sending all those URLs through a 302 and then a 301
- Startupcity.org – has additional 301 redirects leading to that page, in addition to internal links not pointing at the most current page.
I believe it is the combination of these three factors that’s causing this, but my findings are certainly not scientifically conclusive. I always encourage you to do your own questioning and investigating.
So site owners, follow the best practices above, and remember how interesting it is to bring this third service of URL shortening into the mix to contribute to these sorts of issues.
It seems this topic was of much larger discussion around 2009. Understandably so, since its kind of “old news” now. But is it really old news? Danny Sullivan says that it’s really time for an updated look into this topic.
I personally hope this post may get some other people to look more into this topic, starting with the comments. I’d love to hear your thoughts on this topic.
And…
I’ll be at MozCon… come say hi!
or say hi out in the interwebs 🙂
You thought this post was done? Well, so did I! And then I did one last search for startupcity.org – only to find the whole site now redirects to another URL, startupseattle.com
So if you go and try to research this yourself – it seems like this case is changing by the hour! Definitely an interesting one to follow.
